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An Empirical investigation of the Effect of Unequal Variances 
on the Permutation t-test* 

Larry E. Toothaker 
University of Oklahoma 

Various cases of unequal variances and unequal sample sizes from a 
normal and a skewed population were used to empirically obtain the 
probability of a Type I error and the power for the permutation t-test as 
compared to Student's t-test and the Hann-Whi tney U-test. Empirical 
results showed differences for different sample sizes, variance ratios, 
population sampled, and size of mean of the popuiation. The power of the 
permutation t-test is very close to or greater than that of Student’s 
t-test for both populations, and the power is large if the large variance 
accompanied the large mean for the skewed population. 
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An Empirical Investigation of the Effect of Unequal Variances 
on the Permutation t-test 
Larry E. Toothaker 
University of Oklahoma 

The purpose of this research was to investigate empirically the effect 
of unequal variances on a dis cribution-f ree procedure, the permutation t-test, 
for the two sample case. Specifically, the probability of a Type I error 
and the power of the permutation t-test, against a one-tailed location-shift 
alternative hypothesis were empirically obtained when one sample was drawn 
from a population with variance Oj, and the second sample was drawn from a 
population with variance (a^ 1 o|), when both populations were one of 
two types: a normal population and a skewed populatic derived from a x 2 

distribution with three degrees of freedom and skewness measure Yj ~ 1.633. 
Comparisons were then made with the empirical probability of a Type I error 
and the power for Student's t-test and the !Iann-Whi cney U-test. 

The empirical probability of a Type I error and power for the permutation 
t-test, Student’s t-test and the Hann-Whi tney U-test were obtained by means 
of a computer program written to perform the following steps 1) obtain one 
sample from a population with variance and a second sample from a population 
with the same shape as the first bv.t with variance by means of psuedo- 
randorn number generators, 2) add either a null or non-null 9 to one of the 
samples, 3) perform the permutations, rankings and calculations necessary to 
compute each of the three statistics, 4) record either rejection or acceptance 
for each of the statistical procedures, and 5) repeat steps 1-4 for 1000 pairs 
of samples. This procedure resulted in one of the samples being drawn from a 
population with mean \i and a second sample from a population with mean P + 0. 
When 9 B 0, the proportion of rejections was the empirical probability of a 
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Type I error. When each of several non-null values of a 6 (small; medium, or 
large) was used, the nroDortion of rejections was the empirical power. The 
entire procedure was repeated so that results were obtained for all combinations 
of small and large sample sizes with small and large variances and small and 
large means (p or y +0). 

The empirical results for the probability of a Type I error (Table 1) for 
the permutation t-test, Student’s t-test and the fann-Whitney U-test showed 
no consistent relationships among the three statistical procedures. Comparing 
the empirical values of the probability of a Type I error for the permutation 
t-test to those of Student’s t-test gave the following frequencies: pt < t, 

four out of ten, pt > t, four out of ten; and, pt = t, two out of ten- For 
the ifann- Whitney U-test pt < hi three out of ten, pt > g, four out of ten, 
and pt * U. The relation of the empirical values of size for the permutation 
t-test to the theoretical ? was very much dependent upon the sample size, 
ratio of variances and the parent population. For equal sample sizes 
with a normal population, the following empirical values and theoretical 
values were observed for various variance ratios sample sizes (3,3), .046 
and .05 for 1:3; sample sizes (5,5), .076 and .0476 for 1:3; and, sample sizes 
(5,5), .O 54 and .0476 for 1:5. For equal sample sizes with the skewed population, 
the values of the probability of a Type I cr:or were always very much less than 
the theoretical values, for example for sample sizes (5,5), variance ratio 
1:3, the empirical value was .006 for theoretical value .0476. For both populations 
when the sample sizes were unequal, the empirical values were always less than 
the theoretical values if the small variance was with the snail sample size 
(and large variance with large sample size) . When the small variance was with the 
large sample size (and large variance with small sample size), the empirical values 
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were larger than the theoretical values. Sampling from trie skewed population 
merely accomplished a more drastic reduction or inflation of the empirical 
values of probability of Type i error. 

The empirical results for the power of the permutation t-test (Table 2) 
showed interesting comparisons which were dependent upon the population 
sampled. For the normal population with equal sample sizes, the power of the 
permutation t-test is equal to or greater than that of Student^ t-test for 
nine of eighteen cases cud very close for the other nine cases. For unequal 
sample sizes from die normal population, if the small variance was with the 
small sample size (and large variance with large sample size) the power values 
for the permutation t-test were larger than or equal to those for Student's 
t-test for five of the six cases, but if the large variance was with the 
small sample size (and small variance with large sample size) none of the six 
power values for the permutation t-test were larger than or equal to those 
for Student's t-test. In all cases, the power values of the permutation 
t-test were close to those for Student's t-cer.t and larger than or equal to 
those for Lhc liana-Whitaey U-tert. /Iso, there were no consistent differences 
in power values for the procedures with the large mean (y + 6) combined with 
the small sample or th? large sample for the normal population. 

For the skewed population, the most outstanding result was the occurrence 
of very high pov;er values if the large variance accompanied the large mean 
(y + 0) and very low power values if the large variance accompanied the small 
mean (y) for all three statistical procedure?. This result seemed independent 
of the sample sizes employed, equal or unequal. Also, the power values for 
th- permutation t-test were greater than or equal to those of the Student's 
t-test in sixteen of the thirty cares, with the large majority of the cases 
of larger power for the permutation t-test occurring for the small and 
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mediun values of ® . 

Thus, the power of the permutation t-test is generally very close to that of 
Studeiit's t-test when the population variances arc unequcal . When the sample sizes 
also are unequal and proportional to the variances, the power of the permutation 
t-test is better than that of Student's t-test for bcah the normal and skewed 
population. However, if the sample sizes are inversely proportional to the 
/ariances , then Student r s t-test lias higher power than does the permutation t-test, 
especially for the normal population. 

Generally, the present research demonstrated that for the probability of a 
Type I error, the permutation t-test was reasonably close to that of Student's 
t-test, even though both were affected by unequal variances and unequal sample 
sizes, especially for the skewed population. 

The choice of one statistical method over another must be based on consideratic 
of both die probability of a Type I error and the power of the methods. The 
experimental situation must also be considered in terms of population variances, 
the population sampled and the sample sizes obtained. The scientific importance 
of this study ia based on the importance of the need of information on the effect 
of violation of assumptions of non-parame trie statistics which are considered 
substitutes for the carefully researched Student's t-test. Educational researchers 
may now have comparative information on three statistics for the two-sample case 
when normality and equal variances assumptions are violated, thus, they may bo able 
to more correctly choose the statistical method which is best for their experimental 




situation . 



Table 1 



Probability of a Type I Error for the Permutation t-test (pt), 
the .Iann-Uhi tney U-test (U) and Student's t-test (t) for 
Hormal and Skewed Populations with Various 
Ratios of Population Variances 
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Table 2 (Continued) 



